Compressed domain data channel for watermarking, scrambling and steganography

ABSTRACT

A computer-implemented method is provided for processing a video stream in the compressed domain for watermarking, scrambling and other applications. Syntax elements are generated for input video as part of a video compression process. The syntax elements are entropy coded with an arithmetic entropy encoding process to produce a compressed bitstream for the input video. Regions of frames and related syntax elements of the input video are identified as candidates for modification. Based on metadata associated with a particular user, the syntax elements, the regions, and entropy coding state of the arithmetic entropy encoding process, bytes of the input video are changed to generate a modifying bitstream that is unique to the particular user; and modifying the compressed bitstream using the modifying bitstream to produce a decodable bitstream for the input video.

TECHNICAL FIELD

The present disclosure relates to relates to media encoding anddecoding.

BACKGROUND

It is often useful to add a user-specific data channel to a mediastream, such as for watermarking, in order to provide a method foridentifying the source of a copyright breach. Such channels can also beused for steganographic purposes. However, providing a specificwatermark for each user typically involves that the watermark be appliedto the original media prior to or during encoding. As a result, addingnew users involves specific per-user re-encoding. This is not feasiblein practice.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is block diagram of a media distribution system configured toperform watermarking or scrambling of a media stream in a compresseddomain, according to an example embodiment.

FIG. 2 is a high-level flow diagram of operations for watermarking orscrambling a media stream in the compressed domain, according to anexample embodiment.

FIG. 3 is a flow diagram of operations for watermarking or scrambling amedia stream in the compressed domain, according to another exampleembodiment.

FIG. 4 is a flow diagram of operations for watermarking or scrambling amedia stream in the compressed domain, according to yet another exampleembodiment.

FIG. 5 is a flow diagram of operations to generate a modifying stream(microstream) according to an example embodiment.

FIG. 6 is a flow chart of operations for watermarking or scrambling amedia stream in the compressed domain, according to an exampleembodiment.

FIG. 7 is a flow diagram of operations to recover a watermarkrepresented by metadata contained in a received media stream, accordingto an example embodiment.

FIGS. 8 and 9 are flow diagrams of operations to descramble a mediastream that was scrambled according to the techniques presented herein,in accordance with an example embodiment.

FIG. 10 is a hardware block diagram of an apparatus that may beconfigured to perform the encoding-side and decoding-side operationsdescribed herein, according to an example embodiment.

DESCRIPTION OF EXAMPLE EMBODIMENTS Overview

In one embodiment, a computer-implemented method is provided forprocessing a video stream in the compressed domain for watermarking,scrambling and other applications. Syntax elements are generated forinput video as part of a video compression process. The syntax elementsare entropy coded with an arithmetic entropy encoding process to producea compressed bitstream for the input video. Regions of frames of theinput video and related syntax elements are identified as candidates formodification. Based on metadata associated with a particular user, thesyntax elements, the regions, and entropy coding state of the arithmeticentropy encoding process, bytes of the input video are changed togenerate a modifying bitstream that is unique to the particular user.The compressed bitstream is modified using the modifying bitstream toproduce a decodable bitstream for the input video.

Example Embodiments

Presented herein are techniques to modify compressed media bitstreamssuch that a watermark (or other data) can be added in the compresseddomain using lightweight (low cost) methods. Conventional watermarkingtechniques operate in the picture domain before compression, requiringan additional decoding and encoding operation if a compressed stream isto be modified. It is desirable that a watermark be detected in themedia domain by commonly deployed methods.

One example use case is in media broadcast where it is desirable toproduce a watermark for individual streams in a cost effective manner toassist in determining who is the violator of an authorized copy of themedia by virtue of to whom that copy was originally sent or provided.After identifying the violator, the violator's distribution site can betaken down, for example.

Turning now to FIG. 1, a block diagram of a media distribution system100 according to an example embodiment is shown. The system 100 includesa head-end 110 or other distribution device that has access to adatabase of media content 120 for distribution to one or more userdevices 130(1)-130(N) over a distribution network (e.g., the Internet)140. The media content may include video content such as movies or othervideo programs (including software games, etc.). The system 100 isconfigured to be able to either detect the identity (by virtue ofidentifying the particular copy of the media content through a watermarkincluded in the compressed data of the media content) of a source thatdistributed the media content without authorization, and/or to scramblethe compressed media content so as to prevent unauthorized use of themedia content. The user devices 130(1)-130(N) may be any type of device,physical or virtual (entirely-software based), that can decode acompressed media stream for presentation to a user on a display.Examples of such user devices include set-top boxes (STBs), laptopcomputers, desktop computers, Smartphones, tablet computers, softwaremedia players, etc.

A “lightweight” method is provided for generating a set of smallbitstreams, called “microstreams” that may consist of only 10s or 100sof bytes spread across different locations in video framed data, forexample, which can be used to modify a compressed video stream in a waythat can be unique for each user. A microstream is also referred toherein as a “modifying bitstream” because it is ultimately used tomodify a compressed (encoded) bitstream at a decoder (in a user device)in order to produce a valid and decodable bitstream. The compressedvideo stream may be one that is generated according to the H.264 orH.265 video codec standards. The modifications to the compressed videostream can produce changes that are imperceptible for the purposes ofwatermarking or steganography, or perceptible for the purposes ofscrambling and access control. These unique per-user bitstreams can beadded either at a distribution point (e.g., the head-end 110) orsecurely in a user device or other similar device or process (inhardware, hardware and software, or software only). To this end, thehead-end 110 may be configured with compressed domainwatermarking/scrambling logic 150 or each user device 130(1)-130(N) maybe configured with compressed domain watermarking/scrambling logic 150.The compressed domain watermarking/scrambling logic 150 may be embodiedby software instructions executed by a processor, appropriatelyconfigured hardware (digital logic gates), or a combination of softwareand hardware.

Reference is now made to FIG. 2 for an overview of the operations of thecompressed domain watermarking/scrambling logic 150. In particular, FIG.2 shows a flow diagram of a process 200 by which different microstreamsare generated in a compressed-domain watermarking system, according toone example embodiment. The information to be conveyed is embodied inmetadata 210. In other words, the watermarking signal is generated fromthe metadata. While not shown in FIG. 2, it is to be understood that thewatermarking signal may be encoded for protection. One means of encodingis to generate a Pseudo-Random Bitstream Sequence (PRBS) from aspreading key and the watermarking metadata, which has a much higherdata rate than the original watermarking metadata. Forward ErrorCorrection (FEC) codes may also be employed either separately or incombination.

At 220, video encoding syntax generation is performed for input video.This syntax generation may be performed in accordance with the H.264 orH.265 video codec standards, for example, or any other suitable videocodec standard now known or hereinafter developed that has suitableproperties described further below. The metadata 210 representing thewatermarking signal is used to modify syntax elements of the compressedvideo stream. To this end, at 230, the video is analyzed to identifyportions/regions of frames affected by those syntax elements as well asthe related syntax elements. In other words, at 230, regions of framesand related (corresponding) syntax elements of the input video areidentified as candidates for modification. For example, high-activityregions with potential to mask a watermarking signal can be identified,and within those regions, coefficients are identified that could bemodified with low visibility. At 240, using the regions identified at230, the corresponding syntax elements are modified to produce amodified syntax stream. The modified syntax stream is entropy coded at250 to produce a compressed video stream. In addition, an unmodifiedvideo stream is entropy coded at 260.

An arithmetic entropy encoding process is used to encode both themodified syntax elements and the unmodified syntax elements (at 250 and260, respectively) to produce two bitstreams. At 270, the compressedvideo data from the modified syntax elements is compared with thecompressed video data from the unmodified syntax elements to produce alist of differences that are included in a microstream (modifyingbitstream) 275. According to the techniques presented herein, the syntaxelements are bypass-encoded in order to ensure that the microstream canconsist of a small number of isolated bytes (usually 1) for each bitchanged in the syntax.

FIG. 3 shows a process 200′ that is similar to process 200 but in whicha microstream is generated directly without entropy encoding of twodifferent syntax streams. Given knowledge of which syntax elements areto be modified and how, and the current state of the entropy encoderarithmetic encoding process, a microstream generation operation 280 maydirectly determine which bytes will be modified and how they aremodified, which will comprise the microstream 275. The microstreamgeneration operation 280 is described below in more detail in connectionwith FIG. 5. The microstream 275 is then supplied to a bitstreammodification operation 285 that produces a modified compressed videobitstream.

The bitstream modification operation 285 involves modifying thecompressed video bitstream (a normal encoded video bitstream) with themicrostream (modifying bitstream) to produce a valid and decodable(modified) bitstream for the input video. The modified compressed videodata output by operation 285 is a representation of the input video. Inone example, the modified compressed video data is a compressed(encoded) version of the input video with watermarking data added (inthe compressed domain) based on the metadata so as to be visuallyimperceptible to a user when the modified compressed video data isdecoded by a decoder at a user device, but which watermarking data canbe extracted. In another example, the modified compressed video data isa scrambled version (in the compressed) of the input video domain whenthe metadata is a scrambling code. The metadata is also used todescramble the modified compressed video data.

FIG. 4 shows an extension of the process 200′ of FIG. 3, for generatinga plurality of microstreams. Specifically, process 200″ includesoperations 220, 230, 250 and 280, like those shown in FIG. 3, but themicrostream generation operation 280 receives a plurality of sets ofmetadata 210(1)-210(N), where each metadata set is for generation of acorresponding microstream for a particular one of N users. Themicrostream generation operation 280 uses analysis of syntax elementswhich may be modified and the results of these modifications to producea plurality of microstreams (modifying bitstreams) 275(1)-275(N) basedon the plurality of sets of metadata 210(1)-210(N).

More specifically, using a compressed H.264 or H.265 stream “X”, anumber of microstreams 1−N (for N users) are produced, each of which isunique to a corresponding user such that the compressed bitstream Xcombined with any of these microstreams is a valid and decodablebitstream. This may done in a way that is imperceptible to the viewer,but which could be detected by watermarking detectors; or to scramblethe video so that the user needs to have his/her specific microstream inorder to view good-quality video.

As shown in FIG. 4, the bitstream modification operation 285 cangenerate N different modified compressed video bitstreams by modifyingthe compressed video bitstream output by entropy coding operation 250with a corresponding one of the microstreams 275(1)-275(N).

Reference is now made to FIG. 5. FIG. 5 shows a flow diagram for themicrostream generation operation 280. The microstream generationoperation 280 includes a parser/analyser function 300, a locationgenerator function 310 and a data encoding function 320. Theparser/analyser function 300 receives as input syntax elements fromoperation 220 (FIG. 3 or 4) and video region identifiers from operation230 (FIG. 3 or 4) and identifies potential syntax elements formodification. Potential syntax elements are encoded using bypass bits,but may need to satisfy other conditions according to the application.For watermarking applications, these potential syntax elements may bebypass-encoded syntax elements which relate to suitable regions (forexample, high activity regions) of video where metadata may be hidden,in the current frame but also in subsequent frames that depend on thecurrent frame. Examples include least-significant bits of transformcoefficients, or motion vectors, or the signs of small coefficients. Forscrambling applications these may be regions where modifying data wouldhave the most impact and cause the most disruption to the video content.Examples include the signs of significant coefficients or of largemotion vectors. The parser/analyser function 300 then outputs thesesuitable/potential syntax elements to the location generator function310.

The location generator function 310 also receives as input entropycoding state from operation 250 (FIG. 3 or 4) and identifies which byteswould be changed and how, for each suitable/potential syntax elements.For example, this is determined by modifying the entropy coding state atthe point at which the syntax element is encoded so that a “0” isencoded instead of a “1” or vice-versa, and comparing the resulting byteoutput. Each potential syntax location is then associated with amodified byte and byte location. Given a set of metadata to be encodedby the data encoding function 320, the byte locations within the set ofpossible locations are modified according to the unique metadata 210(i)to produce a corresponding microstream 275(i).

Thus, the microstream generation operation 280 takes potential locationsand produces a list of pairs (offset, value) where each offset is alocation that can be modified and each value is a legal value at thatlocation. The pattern of (offset,value) pairs is an encoding of themetadata, and can be made unique to the user or application. In thebitstream modification operation 285 shown in FIGS. 3 and 4, a bytesubstitution operation is made which takes the microstream of(offset,value) pairs and modifies the compressed bitstream to containthe given value at the given offset, thereby producing the modifiedcompressed video bitstream.

The Parser/Analyzer in More Detail

The parser/analyzer 300 is now described in more detail. Theparser/analyzer 300 identifies bytes in the stream that can be modifiedand still produce a legal stream. Apart from packet headers, all data inan H.264 or H.265 stream are encoded with an entropy encoding process.In H.264 this is either an adaptive variable length coder (VLC) such asa context-adaptive variable length coder (CAVLC) or an arithmetic coderusing Context-Adaptive Binary Arithmetic Coding (CABAC). In H265, onlyCABAC is used.

In CABAC, changing any one symbol value changes the arithmetic encoderstate and the contexts, and the entire subsequent bitstream changes.Conversely, changing any single bit in a compressed CABAC stream changesall subsequent decoded values and produces either an illegal stream orcompletely corrupted video. However, there is an exception, and this isthe encoding of binary symbols in a special “bypass” mode. This mode isinvoked when the probabilities are exactly 50:50.

If a bypass-coded value is changed, then due to the internal operationof the arithmetic entropy encoder, a small number of bytes in thebitstream are changed. Normally, this is just one byte, but there may be“carries” where the change in internal state causes preceding bytes tochange. Usually a carry is one byte, occasionally two, and very rarelymore. Therefore, the techniques presented herein rely on modifyingbypass-coded bits in the entropy coded stream. The microstreamgeneration process 280 parses the video stream and identifies suitablevalues to change, and determines what impact changing them has on theencoded video stream.

Bypass-coded bits include:

-   -   sign bits    -   least significant bits (LSBs) of transform coefficients (for        example, suffix bits of coeff_abs_level_minus1 in H.264)

In CAVLC, since actual bits are written, any bits may be modified solong as this does not affect the way contexts are adapted. This alsoincludes sign bits and coefficient LSBs (of Direct Cosine Transform(DCT) coefficients), as well as motion vectors, filter coefficients,mode information (intra prediction modes and inter prediction modes),loop filter parameters, filter coefficients, filter selectionparameters, block modes, or transform modes, as well as metadatacontained in data headers.

Thus, the bypass mode is used to change a small number of bytes, and thebitstream can be “patched-up” in the compressed domain with thedifferent byte values. In this way, it is possible to distribute a largenumber of microstreams to user devices, and the microstreams can be usedat the user devices to modify the bitstream or a user device can decodea bitstream and change a few elements and then recompose the bitstream,and use that as the local copy.

Set forth below are text hexadecimal data dumps of two bitstreams, onewith random bypass bits changed, and the other with random bypass bitsnot changed. In addition, a difference shown between those twobitstreams is provided. The video file is 1 frame of Forman at QuarterCommon Intermediate Format (QCIF) resolution. Both bitstreams representdecodable bitstreams.

No Bypass Bit Modification:

-   0000000 000 000 000 001 147 144 000 050 254 321 202 304 344 000 000    000-   0000010 001 150 351 112 070 260 000 000 000 001 150 132 122 336 054    000-   0000020 000 000 001 150 172 122 356 054 000 000 000 001 145 210 204    000-   0000030 042 211 327 134 372 373 121 164 236 033 162 163 072 261 104    133-   0000040 120 312 030 142 340 262 151 327 156 216 162 211 041 164 144    042-   0000050 372 314 046 021 257 166 025 340 002 044 044 126 352 116 306    261-   0000060 104 331 125 117 366 175 161 140 025 032 121 014 337 274 205    031-   0000070 254 213 354 163 141 105 307 223 052 322 161 265 342 251 037    135-   0000080 360 163 245 164 041 137 254 110 201 323 353 372 025 105 361    226-   0000090 355 135 260 131 257 016 143 277 031 253 166 025 111 124 263    347-   00000a0 172 067 365 351 064 375 075 054 221 137 252 030 301 045 120    352-   00000b0 042 242 060 112 203 341 206 114 233 207 331 016 127 341 120    347-   00000c0 045 325 070 112 222 140 047 147 055 073 114 144 004 362 074    347-   00000d0 345 172 156 373 162 300 115 140 217 015 052 115 207 020 001    251-   00000e0 266 142 121 225 156 064 210 172 230 213 130 027 213 374 267    057-   00000f0 317 144 334 264 272 327 116 363 233 037 146 046 012 267 031    126-   0000100 121 021 273 014 162 333 351 035 073 202 337 223 032 167 117    113-   0000110 236 002 306 041 170 231 062 014 362 341 347 344 236 070 361    016-   0000120 150 252 032 305 044 317 254 344 264 217 111 257 016 137 325    335-   0000130 125 231 017 013 253 342 031 254 101 007 311 003 337 175 110    052-   0000140 230 201 230 276 076 345 064 010 002 214 351 377 334 241 051    323-   0000150 316 173 370 134 034 072 375 246 327 377 101 205 114 245 017    122-   0000160 061 176 267 136 213 157 153 275 255 003 043 005 270 276 120    047-   0000170 106 032 117 060 042 210 254 215 176 306 122 152 127 145 324    361-   0000180 132 244 265 125 074 005 377 214 312 315 230 251 345 344 232    104-   0000190 224 243 314 312 055 304 265 170 123 037 344 137 177 165 040    120-   00001a0 330 035 121 334 047 046 274 005 357 235 064 112 056 017 063    170-   00001b0 206 131 273 326 256 252 111 151 211 177 313 207 304 054 307    347-   00001c0 154 066 021 232 200 331 151 153 022 314 344 304 333 346 332    375-   00001d0 172 016 164 235 316 123 120 257 167 255 272 167 211 176 215    232-   00001e0 246 167 152 205 127 253 211 004 114 024 357 176 045 201 312    275-   00001f0 142 011 302 023 374 030 341 326 310 257 215 211 137 070 124    152-   0000200 105 016 100 063 135 225 104 233 065 254 033 316 362 127 141    203-   0000210 311 320 157 017 031 376 313 012 240 171 343 137 227 272 032    123-   0000220 323 126 156 206 313 122 243 041 047 120 162 127 155 315 011    266-   0000230 052 245 162 335 210 031 362 205 277 123 244 112 150 331 346    272-   0000240 132 363 250 035 070 310-   0000246

Bypass Bit Modification:

-   0000000 000 000 000 001 147 144 000 050 254 321 202 304 344 000 000    000-   0000010 001 150 351 112 070 260 000 000 000 001 150 132 122 336 054    000-   0000020 000 000 001 150 172 122 356 054 000 000 000 001 145 210 204    000-   0000030 042 211 327 134 372 373 121 164 236 033 162 163 072 261 104    133-   0000040 120 312 030 142 340 262 151 327 156 216 162 211 041 164 144    042-   0000050 372 314 046 021 257 166 025 340 002 044 044 126 352 116 306    261-   0000060 104 331 125 117 366 175 161 140 025 032 121 014 337 274 205    031-   0000070 254 213 354 163 141 105 307 223 052 322 161 265 342 251 037    135-   0000080 360 163 245 164 041 137 254 110 201 323 353 372 025 105 361    226-   0000090 355 135 260 131 205 016 143 277 031 253 166 025 111 124 263    347-   00000a0 172 067 365 351 064 375 075 054 221 137 252 030 301 045 120    352-   00000b0 042 242 060 112 203 341 206 114 233 207 331 016 127 341 120    347-   00000c0 045 325 070 112 222 140 047 147 055 073 114 144 004 362 074    347-   00000d0 345 172 156 373 041 300 115 140 217 015 052 115 207 020 001    251-   00000e0 266 142 121 225 156 064 210 172 230 213 130 027 213 374 267    057-   00000f0 317 144 334 264 272 327 116 363 233 037 146 046 012 267 031    126-   0000100 121 021 273 014 162 333 351 035 073 202 337 223 032 167 117    113-   0000110 236 002 306 041 170 231 062 014 362 341 347 344 236 070 361    016-   0000120 150 252 032 305 044 317 254 344 264 217 111 257 016 137 325    335-   0000130 125 231 017 013 253 342 031 254 101 007 311 003 337 175 110    052-   0000140 230 201 230 276 076 345 064 010 002 214 351 377 334 241 051    323-   0000150 316 173 370 134 034 072 375 246 327 377 101 205 114 245 017    122-   0000160 061 176 267 136 213 157 153 275 255 003 043 005 270 276 120    047-   0000170 106 032 117 060 042 210 254 215 176 306 122 152 127 145 324    361-   0000180 132 244 265 125 074 005 377 214 312 315 230 251 345 344 232    104-   0000190 224 243 314 312 055 304 265 170 123 037 344 137 177 165 040    120-   00001a0 330 035 121 334 047 046 274 005 357 235 064 112 056 017 063    170-   00001b0 206 131 273 326 256 252 111 151 211 177 313 207 304 054 307    347-   00001c0 154 066 021 232 200 331 151 153 022 314 344 304 333 346 332    375-   00001d0 172 016 164 235 316 123 120 257 167 255 272 167 211 176 215    232-   00001e0 246 167 152 205 127 253 211 004 114 024 357 176 045 201 312    275-   00001f0 142 011 302 023 374 030 341 326 310 257 215 211 137 070 124    152-   0000200 105 016 100 063 135 225 104 233 065 254 033 316 362 127 141    203-   0000210 311 320 157 017 031 376 313 012 240 171 343 137 227 272 032    123-   0000220 323 126 156 206 313 122 243 041 047 120 162 127 155 315 011    266-   0000230 052 245 162 335 210 031 362 205 277 123 244 112 150 331 346    272-   0000240 132 363 250 035 070 310-   0000246

Difference Bytes:

-   10c10<0000090 355 135 260 131 205 016 143 277 031 253 166 025 111    124 263 347-   --->0000090 355 135 260 131 257 016 143 277 031 253 166 025 111 124    263 347-   14c14<00000d0 345 172 156 373 041 300 115 140 217 015 052 115 207    020 001 251-   --->00000d0 345 172 156 373 162 300 115 140 217 015 052 115 207 020    001 251

Changing the sign of small values, or the LSBs of small values, willproduce small changes in the decoded residual. This will cause smallmismatches with the original reconstructed pictures. This can fit wellwith established watermarking methods, for example, those which modulatevery small changes in colors or DC levels. In other words, forwatermarking applications, these techniques involve making a largenumber of small changes to insignificant portions of the picture, andcorrelating those together. This changes very insignificant parts of thepicture that are difficult to see, which is desired for watermarking.

For scrambling purposes, the modified data can be chosen to have maximumdisruption, for example, if signs of large values or higher level bitsare modified. The metadata is used to repair (unscramble) the bitstream.

To summarize, the microstream generation operation 280 involvesidentifying potential syntax elements for modification based onidentifiers of regions of frames of the input video which are candidatesfor modification; identifying which of the potential syntax elements maybe changed and how bytes of the bitstream may be changed based on theentropy coding state and possible changes to the potential syntaxelements, and producing identified byte locations representativethereof; and modifying the identified byte locations of the bitstreambased on the metadata. In other words, with arithmetic coding a syntaxelement does not directly correspond to a set of bytes in the bitstream.With bypass coding, changes to the value correspond to changes to afinite number of byte locations in the bitstream.

Avoiding Error Propagation

There may be issues of dependencies between frames and within frames tobe addressed in order to avoid error propagation because the decoderattempts to produce decoded video that is the same as the video that theencoder used to encode. Over time these errors may accumulate if thesechanges happen in reference frames and the modified data gets referencedduring prediction.

One way to avoid this is to only modify non-reference frames. Inhierarchical B-coding—the usual approach for broadcast and streamingapplications—half the frames are typically non-reference frames.Alternatively, data can be modified just before an Instantaneous DecoderRefresh (IDR) frame.

In video communications, periodic Gradual Decoder Refreshes (GDRs) areoften used which will reset any accumulated errors. A more sophisticatedsystem could have an encoder create “safe” areas where data can behidden, and the encoder will avoid propagating across many frames.

The principles of binary arithmetic coding are well known. Bit values bhave probabilities p0=p(b=0) and p1=p(b=1), which are known to bothencoder and decoder. The encoder and decoder both maintain asub-interval of [0,1) and adjust this according to the values that areencoded.

If this interval is [Low,Low+Range) then if a 0 is encoded the newinterval becomes:

[Low,Low+Range*p0)

and if a 1 is encoded, the new interval becomes:

[Low+Range*p0,Low+Range)

New values of Low and Range are assigned according to the bounds of thisnew interval. For example, if 0 is coded Low is unchanged and Range isreplaced by p0*Range; if 1 is coded, Low is replaced by Low+p0*Range,and then Range is replaced by p1*Range.

The interval therefore gets smaller and smaller, and the final interval(and any value within it) is unique to the sequence of values that havebeen encoded. Therefore, a binary representation of a value in theinterval is an encoding of the sequence of values.

In a practical encoder, Low and Range and the probabilities arerepresented to finite precision by integer values, and the intervalrescaling by an integer approximation. When the top bit of Low andHigh=Low+Range agree, that bit can be output and all the values rescaledby a factor 2 to keep them in a reasonable range.

There are circumstances in which Range can keep on becoming smaller andthe top bits of Low and High do not agree. In such a case, the encoderwill still rescale Low and stop outputting bits, and can modify pastbits when agreement is eventually reached. This is called a carry.

Bypass Encoding in H.264/H.265

When p0=p1=0.5, the new interval is halved in size. This means thatthere will be a rescale by 2 in the future. H.264 and H.265 instead havea special bypass mode so that the rescale is done straight away, so thatthe range is unchanged and the lower interval bound changed instead:

if value=0, Low->2*Low

if value=1, Low->2*Low+Range

This is very fast to encode and decode compared with the generalarithmetic coding operations.

Modifying Bypass Data

If a 0 or 1 is coded as a bypass value at a given point, then in bothcoding paths, Range is unchanged. In one path, Range has been added toLow and in the other path it has not been added to Low. In H.264/H.265,Range normally has 9 bits (an implementation may use more by maintainingadditional accuracy bits, but this does not affect the output bits ordecoding process). This means that up to 9 bits differ between the twopaths, plus more if coding a 1 caused a carry. Subsequent bits afterthese are the same, as Range is unchanged and Low has finite precision.The carry may cause a chain of carries in preceding bytes, which changestheir values.

Turning now to FIG. 6, a flow chart is shown for a computer-implementedmethod/process 400 according to an example embodiment. The method 400involves generating a microstream (a modifying bitstream) that is uniqueto a particular user, using the techniques described herein. Asexplained above, this process may be performed at a head-end or at auser device, but for purposes of generality, this method 400 isdescribed as being performed at an apparatus. At 410, an apparatusobtains an uncompressed input video for a media asset. At 420, theapparatus generates syntax elements for the input video as part of avideo compression process. At 430, the apparatus entropy codes thesyntax elements with an arithmetic entropy encoding process to produce acompressed bitstream for the input video. At 440, the apparatusidentifies regions of frames of the input video as candidates formodification. At 450, based on metadata associated with a particularuser, the syntax elements, the regions, and entropy coding state of thearithmetic entropy encoding process, the apparatus changes bytes of theinput video to generate a modifying bitstream that is unique to theparticular user. At 460, the apparatus modifies the compressed bitstreamusing the modifying bitstream to produce a decodable bitstream (albeitmodified) for the input video.

In one embodiment, the metadata is descriptive of a watermarking signalunique to the particular user, and in another embodiment, the metadatais descriptive of a scrambling code.

With reference to FIG. 7, a flow chart is shown for a process 500 bywhich the metadata (descriptive of a watermarking signal) is extractedfrom the compressed video that has been generated according to thetechniques described above in connection with FIGS. 1-6. This operationis, in general, performed by an apparatus that has access to themodified compressed bitstream. At 510, the apparatus decodes thecompressed video bitstream using a suitable video decoding processcompliant with any of the aforementioned standards (now known orhereinafter developed) to produce syntax elements and uncompressedvideo. At 520, using the syntax elements and the uncompressed videoobtained at operation 510, the apparatus identifies regions of thevideo, syntax elements, and in particular bypass-coded bits, in whichthe metadata is encoded. At 530, the apparatus performs a correlationdetection operation based on a spreading key 540 to extract the metadatafrom the uncompressed video using the regions identified at 520. Themetadata is, in one embodiment, descriptive of a watermarking signalthat is unique to a particular user. This watermarking signal cantherefore be used to determine the source of the video, that is, forwhom that video was originally intended/authorized.

Reference is now made to FIGS. 8 and 9 for a description of videodescrambling processes useful when a scrambling code has been used tointentionally corrupt video according to the techniques describedherein. Generally, these processes are performed by a user device thatreceives a compressed (and scrambled) bitstream and has a microstream.In FIG. 8, the process 600 uses a microstream to unscramble a scrambledbitstream. At 610, the user device modifies the scrambled bitstreamusing the microstream and an inverse microstream modification operation(inverse to that described above in connection with FIGS. 2-6) toproduce the original (unscrambled) compressed video stream. This inverseprocess invoked at 610 uses the microstream to determine the locationand correct values of the modified bytes, and replaces the modifiedbytes with the correct values. The user device then decodes the original(unscrambled) compressed video stream by a video decoding operation 620to produce a decoded video stream.

FIG. 9 shows a process 700 by which a user device generates an inversemicrostream using the same metadata scrambling key that was used togenerate the original microstream. At 710, the user device decodes thecompressed video stream. At 720, the user device generates themicrostream using the metadata (key) 730 (using a process such as thatshown in FIG. 5). At 740, the user device modifies the compressed videousing the microstream, thereby unscrambling the compressed video toproduce an unscrambled compressed video stream. At 750, the user devicedecodes the unscrambled compressed bitstream to produce a decoded videostream.

FIG. 10 is a block diagram of a computer system 800 upon which theembodiments presented may be implemented. The computer system 800 may beprogrammed to implement a computer based device, such as any device thatincludes a video encoder and/or video decoder that performs theoperations described above in connection with FIGS. 1-9. The computersystem 800 includes a bus 802 or other communication mechanism forcommunicating information, and a processor 803 coupled with the bus 802for processing the information. While the figure shows a single block803 for a processor, it should be understood that the processors 803represent a plurality of processing cores, each of which can performseparate processing. The computer system 800 also includes a main memory804, such as a random access memory (RAM) or other dynamic storagedevice (e.g., dynamic RAM (DRAM), static RAM (SRAM), and synchronousDRAM (SD RAM)), coupled to the bus 802 for storing information andinstructions to be executed by processor 803. In addition, the mainmemory 804 may be used for storing temporary variables or otherintermediate information during the execution of instructions by theprocessor 803.

The computer system 800 further includes a read only memory (ROM) 805 orother static storage device (e.g., programmable ROM (PROM), erasablePROM (EPROM), and electrically erasable PROM (EEPROM)) coupled to thebus 802 for storing static information and instructions for theprocessor 803.

The computer system 800 also includes a disk controller 806 coupled tothe bus 802 to control one or more storage devices for storinginformation and instructions, such as a magnetic hard disk 807 or solidstate drive, and a removable media drive 808 (e.g., floppy disk drive,read-only compact disc drive, read/write compact disc drive, removablemagneto-optical drive and optical storage drive). The storage devicesmay be added to the computer system 800 using an appropriate deviceinterface (e.g., small computer system interface (SCSI), integrateddevice electronics (IDE), enhanced-IDE (E-IDE), direct memory access(DMA), or ultra-DMA), or any other technologies now known or hereinafterdeveloped.

The computer system 800 may also include special purpose logic devices(e.g., application specific integrated circuits (ASICs)) or configurablelogic devices (e.g., simple programmable logic devices (SPLDs), complexprogrammable logic devices (CPLDs), and field programmable gate arrays(FPGAs)), that, in addition to microprocessors and digital signalprocessors may individually, or collectively, are types of processingcircuitry. The processing circuitry may be located in one device ordistributed across multiple devices.

The computer system 800 may also include a display controller 809coupled to the bus 802 to control a display 810, such as a LiquidCrystal Display (LCD), Light Emitting Diode (LED) display, or other nowknown or hereinafter developed display technologies, for displayinginformation to a computer user. The computer system 800 includes inputdevices, such as a keyboard 811 and a pointing device 808, forinteracting with a computer user and providing information to theprocessor 803. The pointing device 808, for example, may be a mouse, atrackball, a pointing stick or a touch-pad, for communicating directioninformation and command selections to the processor 803 and forcontrolling cursor movement on the display 810. The display 810 may be atouch-screen display.

The computer system 800 performs a portion or all of the processingsteps of the process in response to the processor 803 executing one ormore sequences of one or more instructions contained in a memory, suchas the main memory 804. Such instructions may be read into the mainmemory 804 from another computer readable medium, such as a hard disk orsolid state drive 807 or a removable media drive 808. One or moreprocessors in a multi-processing arrangement may also be employed toexecute the sequences of instructions contained in main memory 804. Inalternative embodiments, hard-wired circuitry may be used in place of orin combination with software instructions. Thus, embodiments are notlimited to any specific combination of hardware circuitry and software.

As stated above, the computer system 800 includes at least one computerreadable medium or memory for holding instructions programmed accordingto the embodiments presented, for containing data structures, tables,records, or other data described herein. Examples of computer readablemedia are compact discs, hard disks, floppy disks, tape, magneto-opticaldisks, PROMs (EPROM, EEPROM, flash EPROM), DRAM, SRAM, SD RAM, or anyother magnetic medium, compact discs (e.g., CD-ROM), or any otheroptical medium, punch cards, paper tape, or other physical medium withpatterns of holes, or any other medium from which a computer can read.

Stored on any one or on a combination of non-transitory computerreadable storage media, embodiments presented herein include softwarefor controlling the computer system 800, for driving a device or devicesfor implementing the process, and for enabling the computer system 800to interact with a human user (e.g., print production personnel). Suchsoftware may include, but is not limited to, device drivers, operatingsystems, development tools, and applications software. Such computerreadable storage media further includes a computer program product forperforming all or a portion (if processing is distributed) of theprocessing presented herein.

The computer code devices may be any interpretable or executable codemechanism, including but not limited to scripts, interpretable programs,dynamic link libraries (DLLs), Java classes, and complete executableprograms. Moreover, parts of the processing may be distributed forbetter performance, reliability, and/or cost.

The computer system 800 also includes a communication interface 813coupled to the bus 802. The communication interface 813 provides atwo-way data communication coupling to a network link 814 that isconnected to, for example, a local area network (LAN) 815, or to anothercommunications network 816 such as the Internet. For example, thecommunication interface 813 may be a wired or wireless network interfacecard to attach to any packet switched (wired or wireless) LAN. Asanother example, the communication interface 813 may be an asymmetricaldigital subscriber line (ADSL) card, an integrated services digitalnetwork (ISDN) card or a modem to provide a data communicationconnection to a corresponding type of communications line. Wirelesslinks may also be implemented. In any such implementation, thecommunication interface 813 sends and receives electrical,electromagnetic or optical signals that carry digital data streamsrepresenting various types of information.

The network link 814 typically provides data communication through oneor more networks to other data devices. For example, the network link814 may provide a connection to another computer through a local areanetwork 815 (e.g., a LAN) or through equipment operated by a serviceprovider, which provides communication services through a communicationsnetwork 816. The local network 814 and the communications network 816use, for example, electrical, electromagnetic, or optical signals thatcarry digital data streams, and the associated physical layer (e.g., CAT5 cable, coaxial cable, optical fiber, etc.). The signals through thevarious networks and the signals on the network link 814 and through thecommunication interface 813, which carry the digital data to and fromthe computer system 800 maybe implemented in baseband signals, orcarrier wave based signals. The baseband signals convey the digital dataas unmodulated electrical pulses that are descriptive of a stream ofdigital data bits, where the term “bits” is to be construed broadly tomean symbol, where each symbol conveys at least one or more informationbits. The digital data may also be used to modulate a carrier wave, suchas with amplitude, phase and/or frequency shift keyed signals that arepropagated over a conductive media, or transmitted as electromagneticwaves through a propagation medium. Thus, the digital data may be sentas unmodulated baseband data through a “wired” communication channeland/or sent within a predetermined frequency band, different thanbaseband, by modulating a carrier wave. The computer system 800 cantransmit and receive data, including program code, through thenetwork(s) 815 and 816, the network link 814 and the communicationinterface 813. Moreover, the network link 814 may provide a connectionthrough a LAN 815 to a mobile device 817 such as a personal digitalassistant (PDA) laptop computer, or cellular telephone.

The techniques presented herein are intended to be agnostic to anyparticular watermarking (or scrambling) scheme. Again, these techniquesconcern applying a number of different user-specific watermarks bypatching bitstreams in the compressed domain.

To summarize, presented herein is a system for generating a large numberof compliant video streams from a single base stream in the compresseddomain, for watermarking, scrambling or data hiding purposes. Eachmodified stream can be chosen to differ in a small number of bytes fromthe original stream, and the byte pattern can be made to be unique. Themodified stream can have imperceptible video changes.

Adding watermark data (or scrambling) in the compressed domain isparticularly advantageous for its light weight computationalrequirements and for its increased convenience in the video workflow, incontrast with methods operating in the uncompressed domain. Morespecifically, this approach does not involve full re-encoding ofbitstreams, and can generate a large number of unique streams after asingle analysis pass. The “microstreams” added can easily be storedwithin secure storage on a user device, or sent out-of-band during abroadcast.

In summary, in one form, a computer-implemented method is providedcomprising: obtaining uncompressed input video; generating syntaxelements for the input video as part of a video compression process;entropy coding the syntax elements with an arithmetic entropy encodingprocess to produce a compressed bitstream for the input video;identifying regions of frames and related syntax elements of the inputvideo which are candidates for modification; based on metadataassociated with a particular user, the syntax elements, the regions, andentropy coding state of the arithmetic entropy encoding process,changing bytes of the input video to generate a modifying bitstream thatis unique to the particular user; and modifying the compressed bitstreamusing the modifying bitstream to produce a decodable bitstream for theinput video.

In another form, an apparatus is provided comprising: a communicationinterface; and a processor coupled to the communication interface,wherein the processor is configured to: obtain uncompressed input video;generate syntax elements for the input video as part of a videocompression process; entropy code the syntax elements with an arithmeticentropy encoding process to produce a compressed bitstream for the inputvideo; identify regions of frames and related syntax elements of theinput video which are candidates for modification; and based on metadataassociated with a particular user, the syntax elements, the regions, andentropy coding state of the arithmetic entropy encoding process, changebytes of the input video to generate a modifying bitstream that isunique to the particular user; and modify the compressed bitstream usingthe modifying bitstream to produce a decodable bitstream for the inputvideo.

In still another form, one or more non-transitory computer readablestorage media are provided encoded with instructions that, when executedby a processor, cause the processor to perform operations including:obtaining uncompressed input video; generating syntax elements for theinput video as part of a video compression process; entropy coding thesyntax elements with an arithmetic entropy encoding process to produce acompressed bitstream for the input video; identifying regions of framesand related syntax elements of the input video which are candidates formodification; based on metadata associated with a particular user, thesyntax elements, the regions, and entropy coding state of the arithmeticentropy encoding process, changing bytes of the input video to generatea modifying bitstream that is unique to the particular user; andmodifying the compressed bitstream using the modifying bitstream toproduce a decodable bitstream for the input video.

The above description is intended by way of example only. Although thetechniques are illustrated and described herein as embodied in one ormore specific examples, it is nevertheless not intended to be limited tothe details shown, since various modifications and structural changesmay be made within the scope and range of equivalents of the claims.

What is claimed is:
 1. A computer-implemented method comprising:obtaining uncompressed input video; generating syntax elements for theinput video as part of a video compression process; entropy coding thesyntax elements with an arithmetic entropy encoding process to produce acompressed bitstream for the input video; identifying regions of framesand related syntax elements of the input video which are candidates formodification; based on metadata associated with a particular user, thesyntax elements, the regions, and entropy coding state of the arithmeticentropy encoding process, changing bytes of the input video to generatea modifying bitstream that is unique to the particular user; andmodifying the compressed bitstream using the modifying bitstream toproduce a decodable bitstream for the input video.
 2. The method ofclaim 1, wherein the metadata is descriptive of a watermarking signalunique to the particular user.
 3. The method of claim 1, whereinchanging bytes of the input video is performed to produce a uniquemodifying bitstream for each of a plurality of users based on uniquemetadata associated with each of the plurality of users.
 4. The methodof claim 1, wherein changing includes: identifying potential syntaxelements for modification based on identifiers of the regions;identifying which of the potential syntax elements may be changed andhow bytes of the bitstream may be changed based on the entropy codingstate and possible changes to the potential syntax elements, andproducing identified byte locations representative thereof; andmodifying the identified byte locations of the bitstream based on themetadata.
 5. The method of claim 4, wherein the potential syntaxelements which are changed are encoded with bypass-coded bits.
 6. Themethod of claim 5, wherein the modifying includes changing at least oneof sign bits, transform coefficient bits, motion vectors, intraprediction modes, inter prediction modes, filter coefficients or filterselection parameters, loop filter parameters, quantization indices,header data or metadata contained in headers.
 7. The method of claim 1,wherein changing comprises changing bytes of non-reference frames,including frames just before a decoder refresh frame.
 8. The method ofclaim 1, wherein the metadata is descriptive of a scrambling code, andwherein modifying comprises modifying the compressed bitstream with themodifying bitstream to produce a scrambled bitstream based on thescrambling code, and further comprising: sending the scrambled bitstreamto a user device associated with the particular user.
 9. The method ofclaim 8, further comprising: modifying the scrambled bitstream using themodifying bitstream to produce an unscrambled compressed bitstream; anddecoding the unscrambled compressed bitstream to recover the inputvideo.
 10. The method of claim 9, further comprising, at the userdevice: obtaining the metadata; and generating the modifying bitstreamusing the metadata.
 11. An apparatus comprising: a communicationinterface; and a processor coupled to the communication interface,wherein the processor is configured to: obtain uncompressed input video;generate syntax elements for the input video as part of a videocompression process; entropy code the syntax elements with an arithmeticentropy encoding process to produce a compressed bitstream for the inputvideo; identify regions of frames and related syntax elements of theinput video which are candidates for modification; and based on metadataassociated with a particular user, the syntax elements, the regions, andentropy coding state of the arithmetic entropy encoding process, changebytes of the input video to generate a modifying bitstream that isunique to the particular user; and modify the compressed bitstream usingthe modifying bitstream to produce a decodable bitstream for the inputvideo.
 12. The apparatus of claim 11, wherein the metadata isdescriptive of a watermarking signal unique to the particular user. 13.The apparatus of claim 11, wherein the processor is configured to changebytes of the input video to produce a unique modifying bitstream foreach of a plurality of users based on unique metadata associated witheach of the plurality of users.
 14. The apparatus of claim 11, whereinthe processor is configured to change bytes by: identifying potentialsyntax elements for modification based on identifiers of the regions;identifying which of the potential syntax elements may be changed andhow bytes of the bitstream may be changed based on the entropy codingstate and possible changes to the potential syntax elements, andproducing identified byte locations representative thereof; andmodifying the identified byte locations of the bitstream based on themetadata.
 15. The apparatus of claim 14, wherein the potential syntaxelements which are changed are encoded with bypass-coded bits.
 16. Oneor more non-transitory computer readable storage media encoded withinstructions that, when executed by a processor, cause the processor toperform operations including: obtaining uncompressed input video;generating syntax elements for the input video as part of a videocompression process; entropy coding the syntax elements with anarithmetic entropy encoding process to produce a compressed bitstreamfor the input video; identifying regions of frames and related syntaxelements of the input video which are candidates for modification; basedon metadata associated with a particular user, the syntax elements, theregions, and entropy coding state of the arithmetic entropy encodingprocess, changing bytes of the input video to generate a modifyingbitstream that is unique to the particular user; and modifying thecompressed bitstream using the modifying bitstream to produce adecodable bitstream for the input video.
 17. The non-transitory computerreadable storage media of claim 16, wherein the metadata is descriptiveof a watermarking signal unique to the particular user.
 18. Thenon-transitory computer readable storage media of claim 16, wherein theinstructions operable for changing bytes comprises instructions operablefor changing bytes of the input video to produce a unique modifyingbitstream for each of a plurality of users based on unique metadataassociated with each of the plurality of users.
 19. The non-transitorycomputer readable storage media of claim 16, wherein the instructionsoperable for changing bytes comprises instructions operable for:identifying potential syntax elements for modification based onidentifiers of the regions; identifying which of the potential syntaxelements may be changed and how bytes of the bitstream may be changedbased on the entropy coding state and possible changes to the potentialsyntax elements, and producing identified byte locations representativethereof; and modifying the identified byte locations of the bitstreambased on the metadata.
 20. The non-transitory computer readable storagemedia of claim 16, wherein the potential syntax elements which arechanged are encoded with bypass-coded bits.